The applicant of this application previously proposed classification adaptive processing as data conversion processing for improving the quality of images or performing other types of image conversion.
The classification adaptive processing includes classification processing and adaptive processing: data is classified by classification processing according to the property of the data, and each class of the data is subjected to adaptive processing. The adaptive processing is, for example, as follows.
In the adaptive processing, for example, low-quality or standard-quality image (hereinafter sometimes referred to as an “SD (Standard Definition) image”) data is mapped by using predetermined tap coefficients so as to be converted into high-quality image (hereinafter sometimes referred to as an “HD (High Definition) image”) data.
It is now assumed that, for example, a linear coupling model is employed as the mapping method using tap coefficients. In this case, the pixel values of pixels y forming HD image data (hereinafter sometimes referred to as “HD pixels”) are determined by using tap coefficients and a plurality of pixels forming SD image data (hereinafter sometimes referred to as “SD pixels”) extracted as predictive taps for predicting the HD pixels according to the following linear equation (linear coupling).
                    y        =                              ∑                          n              =              1                        N                    ⁢                                    w              n                        ⁢                          x              n                                                          (        1        )            
In equation (1), xn indicates the pixel value of the n-th pixel of the SD image data forming the predictive taps for the HD pixel y, and wn indicates the n-th tap coefficient to be multiplied with the pixel value of the n-th SD pixel. In equation (1), it is assumed that the predictive taps consist of N SD pixels x1, x2, . . . , xN.
The pixel value y of the HD pixel may be determined by equations of higher degrees, such as a quadratic equation, rather than by the linear equation expressed in (1).
When the true value of the pixel value of the k-th sample HD pixel is indicated by yk, and when the predictive value of the true value yk determined by equation (1) is indicated by yk′, the predictive error ek is expressed by the following equation.ek=yk−yk′  (2)
Since the predictive value yk′ in equation (2) is determined by equation (1), equation (1) is substituted into yk′ in equation (2), thereby obtaining the following equation.
                              e          k                =                              y            k                    -                      (                                          ∑                                  n                  =                  1                                N                            ⁢                                                w                  n                                ⁢                                  x                                      n                    ,                    k                                                                        )                                              (        3        )            
In equation (3), xn,k designates the n-th SD pixel forming the predictive taps for the k-th sample HD pixel.
The tap coefficient wn that sets the predictive error ek to be 0 in equation (3) is the optimal value for predicting the HD pixel. Generally, however, it is difficult to determine such tap coefficients wn for all the HD pixels.
Accordingly, as the standard for the optimal tap coefficient wn, the method of least squares, for example, is used. Then, the optimal tap coefficient wn can be determined by minimizing the sum E of the square errors as the statistical error expressed by the following equation.
                    E        =                              ∑                          k              =              1                        K                    ⁢                      e            k            2                                              (        4        )            
In equation (4), K indicates the number of set samples of the HD pixel yk and the SD pixels x1,k, x2,k, . . . , xN,k forming the predictive taps for the HD pixel yk.
The tap coefficient wn that minimizes the sum E of the square errors in equation (4) must satisfy the condition that the value determined by partial-differentiating the sum E with the tap coefficient wn becomes 0, and thus, the following equation must be established.
                                          ∂            E                                ∂                          w              n                                      =                                                            e                1                            ⁢                                                ∂                                      e                    1                                                                    ∂                                      w                    n                                                                        +                                          e                2                            ⁢                                                ∂                                      e                    2                                                                    ∂                                      w                    n                                                                        +            …            +                                          e                k                            ⁢                                                ∂                                      e                    n                                                                    ∂                                      w                    n                                                                                =                      0            ⁢                                                  ⁢                          (                                                n                  =                  1                                ,                2                ,                …                ⁢                                                                  ,                N                            )                                                          (        5        )            
Accordingly, by partial-differentiating equation (3) with the tap coefficient wn, the following equation can be found.
                                                        ∂                              e                k                                                    ∂                              w                1                                              =                      -                          x                              1                ,                k                                                    ,                                            ∂                              e                k                                                    ∂                              w                2                                              =                      -                          x                              2                ,                k                                                    ,        …        ⁢                                  ,                                            ∂                              e                k                                                    ∂                              w                N                                              =                      -                          x                              N                ,                k                                                    ,                                  ⁢                  (                                    k              =              1                        ,            2            ,            …            ⁢                                                  ,            K                    )                                    (        6        )            
The following equation can be found from equations (5) and (6).
                                                        ∑                              k                =                1                            K                        ⁢                                          e                k                            ⁢                              x                                  1                  ,                  k                                                              =          0                ,                                            ∑                              k                =                1                            K                        ⁢                                          e                k                            ⁢                              x                                  2                  ,                  k                                                              =          0                ,                              …            ⁢                                                  ⁢                                          ∑                                  k                  =                  1                                K                            ⁢                                                e                  k                                ⁢                                  x                                      N                    ,                    k                                                                                =          0                                    (        7        )            
By substituting equation (3) into ek in equation (7), equation (7) can be expressed by the normal equations expressed by equations (8).
                                          [                                                                                (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    ⋯                                                                      (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                                                                    (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    ⋯                                                                      (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                                                ⋮                                                  ⋮                                                  ⋰                                                  ⋮                                                                                                  (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    ⋯                                                                      (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                        ]                    [                                          ⁢                                                                      w                  1                                                                                                      w                  2                                                                                    ⋮                                                                                      w                  N                                                              ]                ⁢                                  ⁢                                                            =                                                                    =                                                                    =                                              [                                          ⁢                                                                      (                                                            ∑                                              k                        =                        1                                            K                                        ⁢                                                                  x                                                  1                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                                                                      (                                                            ∑                                              k                        =                        1                                            K                                        ⁢                                                                  x                                                  2                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                                                    ⋮                                                                                      (                                                            ∑                                              k                        =                        1                                            K                                        ⁢                                                                  x                                                  N                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                              ⁢                                          ]                                    (        8        )            
By preparing a certain number of sets of the HD pixels yk and the SD pixels xn,k, the same number of normal equations (8) as the number of tap coefficients wn to be determined can be found, and by solving equations (8) (the matrix at the left side next to the tap coefficients wn in equations (8) must be regular to solve equations (8), the optimal tap coefficients wn can be determined. In solving equations (8), the sweep-out method (Gauss-Jordan elimination), for example, may be employed.
As described above, by solving equations (8) by setting many HD pixels y1, y2, . . . , yK to be supervisor data as supervisors for learning tap coefficients and by setting SD pixels x1,k, x2,k, . . . , xN,k forming the predictive taps for each HD pixel yk to be learner data as learners for learning the tap coefficients, learning is conducted for determining the optimal tap coefficients wn. By using the optimal tap coefficients wn, SD image data is mapped (converted) onto (into) HD image data by using equation (1). The above-described processing is adaptive processing.
The adaptive processing is different from mere interpolation processing in that components contained not in SD images but in HD images are reproduced. More specifically, only from equation (1), the adaptive processing is similar to the so-called “interpolation processing” using interpolation filters. However, the tap coefficients wn, which correspond to tap coefficients used in the interpolation filters, are determined by learning by using HD image data as supervisor data and SD image data as learner data. Thus, components contained in HD images can be reproduced. Accordingly, it is possible that the adaptive processing serves the function of creating images (creating the resolution).
In learning tap coefficients wn, the combinations of supervisor data y and learner data x can be changed so as to obtain tap coefficients wn performing various conversions.
If HD image data is used as the supervisor data y, and if SD image data determined by adding noise or blurring to the HD image data is used as the learner data x, tap coefficients wn for converting an image into an image without noise or blurring can be obtained. If HD image data is used as the supervisor data y, and if SD image data determined by decreasing the resolution of the HD image data is used as the learner data x, tap coefficients wn for converting an image into an image having improved resolution can be obtained. If image data is used as the supervisor data y, and if DCT (Discrete Cosine Transform) coefficients determined by performing DCT on the image data is used as the learner data x, tap coefficients wn for converting the DCT coefficients into image data can be obtained.
As described above, in classification adaptive processing, the tap coefficient wn that minimizes the sum E of the square errors in equation (4) is determined for each class, and equation (1) is calculated by using the tap coefficient wn, thereby converting an SD image into a high-quality HD image. That is, by using the tap coefficients wn and the predictive taps xn generated by the SD image, equation (1) is calculated so as to determine HD pixels forming the HD image.
Accordingly, in the classification adaptive processing previously proposed, when focusing on each HD pixel, the predictive value that statistically minimizes the predictive error of each HD pixel with respect to the true value can be determined.
More specifically, it is now assumed, as shown in FIG. 1A, that there are two HD pixels yk and yk+1, which are horizontally, vertically, or obliquely adjacent to each other. In this case, as for the HD pixel yk, the predictive value yk′ that can statistically minimize the predictive error ek with respect to the true value yk can be obtained. Similarly, as for the HD pixel yk+1 the predictive value yk+1′ that can statistically minimize the predictive error ek+1 with respect to the true value yk+1 can be obtained.
In the classification adaptive processing previously proposed, however, for the two HD pixels yk and yk+1, when the true HD pixel yk obliquely increases to the right side toward the true HD pixel yk+1, as those shown in FIG. 1A, the following result may be sometimes brought about. As shown in FIG. 1B, as for the HD pixel yk, the predictive value yk′, which is greater than the true value, is obtained, and on the other hand, as for the HD pixel yk+1 the predictive value yk+1′, which is smaller than the true value, is obtained.
In this case, the predictive value yk′ for the HD pixel yk decreases to the right side toward the predictive value yk+1′ for the HD pixel yk+1, as shown in FIG. 1B.
As discussed above, if a predictive value decreases to the right side in spite of the fact that a true value increases to the right side, i.e., if a change in the pixel value becomes opposite to a change in the true value, the resulting image quality may be decreased.